Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus capable of easily generating a two-dimensional panoramic image at a high speed from a plurality of three-dimensional images includes an acquisition unit configured to acquire a generation condition of a first en-face image generated from a first three-dimensional image of an target eye, a first generation unit configured to generate a second en-face image from a second three-dimensional image of the target eye by applying the generation condition acquired by the acquisition unit to the second three-dimensional image, and a second generation unit configured to generate a combined image by combining the first en-face image with the second en-face image.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, imageprocessing method, and a storage medium.

Description of the Related Art

As an apparatus for observing a target eye (i.e., eye to be examined),an optical Coherence Tomography (OCT) is known. Further, it is knownthat a two-dimensional image is generated from a three-dimensional imageacquired by using the OCT. The two-dimensional image is called anen-face image obtained by projecting, in a depth direction, pixels in arange located between any two reference surfaces different in a depthposition. Further, Japanese Patent Application Laid-Open No. 2017-47113discusses generation of a two-dimensional panoramic image by combining aplurality of en-face images acquired by using an OCT angiography (OCTA).

However, generation of a two-dimensional combined image such as atwo-dimensional panoramic image places a burden on a user. This isbecause in order to generate a plurality of two-dimensional images assources of the combined image, it is necessary that, for example, theuser sets generation conditions of the two-dimensional images for aplurality of three-dimensional images, such as a reference surface.

The present disclosure is directed to a technique for easily generatinga two-dimensional combined image from a plurality of three-dimensionalimages.

Not limited to the above object, the present disclosure includesoperations and effects that are to be derived by configurations inexemplary embodiments of the present invention described below andcannot be obtained by conventional techniques.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an image processingapparatus includes an acquisition unit configured to acquire ageneration condition of a first en-face image generated from a firstthree-dimensional image of a target eye, a first generation unitconfigured to generate a second en-face image from a secondthree-dimensional image of the target eye by applying the generationcondition acquired by the acquisition unit to the secondthree-dimensional image, and a second generation unit configured togenerate a combined image by combining the first en-face image with thesecond en-face image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a display screen of animage processing apparatus according to a first exemplary embodiment.

FIG. 2 is a diagram illustrating an example of a configuration of animage processing system according to the first exemplary embodiment.

FIG. 3 is a diagram illustrating an example of an image acquisitionmethod according to the first exemplary embodiment.

FIG. 4 is a diagram illustrating an example of a method for generatingan en-face image according to the first exemplary embodiment.

FIG. 5 is a block diagram illustrating an example of a configuration ofthe image processing apparatus according to the first exemplaryembodiment.

FIG. 6 is a flowchart illustrating an example of an operation of theimage processing apparatus according to the first exemplary embodiment.

FIG. 7 is a diagram illustrating an example of a display screen of theimage processing apparatus according to the first exemplary embodiment.

FIG. 8 is a diagram illustrating an example of a display screen of theimage processing apparatus according to the first exemplary embodiment.

FIG. 9 is a diagram illustrating an example of an operation of an imageprocessing apparatus according to a second exemplary embodiment.

FIG. 10 is a diagram illustrating an example of a display screen of animage processing apparatus according to a fourth exemplary embodiment.

FIG. 11 is a flowchart illustrating an example of an operation of theimage processing apparatus according to the fourth exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments will be described below with reference to thedrawings. The same or similar components, members, and processingillustrated in the drawings are denoted by the same numerals or symbols,and overlapped description is suitably omitted. Further, in thedrawings, components, members and processing that are not important forthe description are partially omitted in some cases. Each of theembodiments of the present invention described below can be implementedsolely or as a combination of a plurality of the embodiments or featuresthereof where necessary or where the combination of elements or featuresfrom individual embodiments in a single embodiment is beneficial.

FIG. 2 is a diagram illustrating an example of a configuration of animage processing system according to a first exemplary embodiment. Theimage processing system according to the first exemplary embodimentincludes an optical coherence tomography (OCT) image capturing apparatus201, an image processing apparatus 209, and a display apparatus 101. InFIG. 2, the OCT image capturing apparatus 201, the image processingapparatus 209, and the display apparatus 101 are separated, but theconfiguration is not limited to this form, and for example, the imageprocessing apparatus 209 and the display apparatus 101 may be configuredto be integrated. Further, the OCT image capturing apparatus 201 and theimage processing apparatus 209 may be configured to be integrated.

The OCT image capturing apparatus 201 acquires a signal representing atomographic image of a target eye (i.e., eye to be examined) E based ona two-dimensional measurement range specified on, for example, an eyefundus surface of the target eye E and measurement depth information.The OCT image capturing apparatus 201 is, for example, a spectral-domainOCT (SD-OCT), but any OCT system can be employed as long as the OCT cancapture an image of a fault of the target eye E. As illustrated in FIG.2, the OCT image capturing apparatus 201 includes a light source 202, ahalf mirror 203, a galvano mirror 204, an object lens 205, a referencemirror 206, a diffractive grating 207, and a line sensor 208.

Low coherence light emitted from the light source 202 is divided intomeasurement light and reference light by the half mirror 203.Measurement light enters a target eye E via the galvano mirror 204 andthe object lens 205. A scanning position of the target eye E can bechanged by driving the galvano mirror 204. In FIG. 2, forsimplification, only the one galvano mirror 204 is illustrated, butactually two galvano mirrors that can scan light in mutually orthogonaldirections are used. The measurement light that has entered the targeteye E is reflected and scattered by the target eye E. Then, themeasurement light then returns to the half mirror 203 along a reverseoptical path. The reference light is reflected by the reference mirror206. Then, the reference light returns to the half mirror 203 along areverse optical path. The half mirror 203 overlaps a return light of themeasurement light and a return light of the reference light to generateinterference light. The diffractive grating 207 disperses theinterference light into wavelength components from wavelengths λ1 to λn.The line sensor 208 detects the dispersed interference light for each ofthe wavelength components. The line sensor 208 outputs signal accordingto a detected result.

The OCT image capturing apparatus 201 according to the present exemplaryembodiment has a panoramic imaging mode as a mode for sequentiallyacquiring tomographic images of different portions of the target eye E.In a case where an operator selects this mode, after the imaging of apartial region 301 of the target eye E illustrated in FIG. 3 iscompleted, for example, a visual line of the target eye E is guided to adifferent direction by moving an internal fixation lamp. The imaging inthis state makes it possible to acquire a tomographic image of a partialregion 302 different from the tomographic image of the partial region301. A tomographic image of a residual partial region 303 can beacquired by sequentially executing the similar processing. In a casewhere a wide-angle panoramic combined image based on the tomographicimage captured in the panoramic imaging mode is generated, it isdesirable to move the internal fixation lamp so that adjacent regions(e.g., the partial region 301 and the partial region 302) are at leastpartially overlapped with each other. In the present exemplaryembodiment, the internal fixation lamp is moved so that the partialregion 301 and the partial region 302 are overlapped with each other by25% of areas. The numerical value is an example and thus is not limitedto the example.

The image processing apparatus 209 is, for example, a computer. Theimage processing apparatus 209 generates a tomographic image of, forexample, an eye fundus of the target eye E by using Fouriertransformation, based on an output signal from the line sensor 208.Since the method for generating a tomographic image can be realized by aknown method, detailed description thereof is omitted.

Emission of measurement light to any point of the target eye E isreferred to as “A scan” in the present exemplary embodiment, and atomographic image generated by the A scan is referred to as an “A-scanimage” in some cases. Further, intermittent emission of measurementlight to the target eye E along any scanning line is referred to as “Bscan”, and a tomographic image acquired by the B scan is referred to asa “B-scan image” in some cases. In this way, the B-scan image isconfigured by a plurality of A-scan images.

The image processing apparatus 209 acquires B-scan images of a pluralityof portions of the target eye E to be able to configure athree-dimensional image of the target eye E. A B-scan direction may be aradial scan direction where a specific portion is radially scanned, or ahorizontal scan direction or a vertical scan direction where a specificportion is scanned in a certain one direction.

The image processing apparatus 209 generates, from a three-dimensionalimage, an en-face image that is a two-dimensional image projected to aplane based on any two reference surfaces in a depth direction (Zdirection), for example. FIG. 7 illustrates an example of a B-scan image130 and an en-face image 140 that are generated by the image processingapparatus 209. The en-face image 140 is an image acquired by projecting,to the Z direction, a range lying between a reference surface 130 a anda reference surface 130 b of the B-scan image 130. Specifically, asillustrated in FIG. 4, the image processing apparatus 209 extractsinformation in only a specific depth range of a three-dimensional image(not illustrated), structured from a plurality of acquired B-scan images40-1 to 40-n. The image processing apparatus 20 then projects theinformation to an X-Y plane to generate the en-face image 140. Since anen-face image can be generated by using various known methods, detaileddescription thereof is omitted.

An example of a configuration of the image processing apparatus 209according to the present exemplary embodiment will be described belowwith reference to FIG. 5. The image processing apparatus 209 includes atomographic image acquisition unit 501, a three-dimensional imagegeneration unit 502, a combination target selection unit 503, an en-faceimage generation unit 504, a two-dimensional combined image generationunit 505, and a display control unit 510.

The image processing apparatus 209 includes a central processing unit(CPU) and a read-only memory (ROM) (not illustrated). The CPU executes aprogram stored in the ROM to function as the above-described respectiveunits. The image processing apparatus 209 may include one CPU and oneROM or a plurality of CPUs and ROMs. In other words, in a case where atleast one or more processors and at least one or more memories areconnected, and the at least one or more processors execute programsstored in the at least one or more memories, the image processingapparatus 209 functions as the above-described respective units. Theprocessor is not limited to the CPU, and may be a graphics processingunit (GPU), or different types of processors such as the CPU and the GPUmay be used in combination.

The tomographic image acquisition unit 501 acquires tomographic imagesof a plurality of portions captured by the OCT image capturing apparatus201. The tomographic image acquisition unit 501 may generate atomographic image based on an output from the line sensor 208 to acquirethe tomographic image. Alternatively, the tomographic image acquisitionunit 501 may acquire a tomographic image that has been already generatedfrom a database (not illustrated).

The three-dimensional image generation unit 502 generates athree-dimensional image based on the tomographic images of the pluralityof portions acquired by the tomographic image acquisition unit 501. Thethree-dimensional image generation unit 502, for example, arrange thetomographic images of the plurality of portions on one coordinate systemto generate a three-dimensional tomographic image. Further, thethree-dimensional image generation unit 502 generates a motion contrastimage (OCT angiography image) from a plurality of the tomographic imagesacquired for each portion, and arrange the motion contrast imagesgenerated for each portion on one coordinate system. In such a manner,the three-dimensional image generation unit 502 can also generate athree-dimensional motion contrast image. In other words, thethree-dimensional image generation unit 502 generates athree-dimensional tomographic image (luminance tomographic image) or athree-dimensional motion contrast image as a three-dimensional image.Further, the en-face image generation unit 504 (described below)generates an en-face image of luminance or an en-face image of motioncontrast as an en-face image from this three-dimensional image.

The combination target selection unit 503 selects a combination targetfor generating a two-dimensional combined image. The combination targetmay be automatically selected or may be selected in accordance with aninstruction from an operator. The combination target selection unit 503selects, for example, a three-dimensional image as the combinationtarget.

The en-face image generation unit 504 sets any two surfaces at differentdepth positions to the three-dimensional image. The any two surfaces arereference surfaces when the en-face image is generated. The en-faceimage generation unit 504 generates the en-face image based on a regionlying between the set two reference surfaces. The reference surfaces maybe surfaces along a layer boundary included in the tomographic image ormay be planes. For example, the en-face image generation unit 504determines a representative value of pixel values in the depth directionat positions in the region lying between the two reference surfaces, andgenerates the en-face image based on the representative value. Therepresentative value is an average value, a median value, or a maximumvalue of pixels in the depth direction of the region lying between thetwo reference surfaces. The en-face image generation unit 504 applies auniform generation condition in the processing to all three-dimensionalimages of a plurality of combination targets selected by the combinationtarget selection unit 503. In such a manner, a plurality of en-faceimages is acquired.

The two-dimensional combined image generation unit 505 aligns andcombines the plurality of en-face images generated by the en-face imagegeneration unit 504 to generate a two-dimensional combined image. Thetwo-dimensional combined image generation unit 505 can perform thealignment using, for example, features of blood vessels included in theplurality of en-face images. Further, the two-dimensional combined imagegeneration unit 505 can align the plurality of en-face images based on alighting position of the fixation lamp during the capturing of thethree-dimensional images as sources of the en-face images. Further, thetwo-dimensional combined image generation unit 505 may align theplurality of en-face image using the features of a blood vessel and alighting position of the fixation lamp.

The display control unit 510, for example, causes the display apparatus101 to display the generated two-dimensional combined image.

The display apparatus 101 is a liquid crystal display (LCD) apparatus orthe like, and displays various information based on control by thedisplay control unit 510. For example, as illustrated in FIG. 1, thedisplay apparatus 101 displays an integrating image acquired byintegrating pixel values of the three-dimensional images acquired by thecapturing in the Z direction (depth direction) and a B-scan image in thedisplay area 120. Further, the display apparatus 101 displays atwo-dimensional combined image generated by the two-dimensional combinedimage generation unit 505 in a display area 111. An additionallyprovided optical system may display a scanning laser ophthalmoscope(SLO) image or the like captured simultaneously with the respectivethree-dimensional images and displayed in the display area 120 or anarea that is not illustrated herein.

An example of an operation method and an operation of the imageprocessing apparatus 209 will be described below with reference to FIG.6 to FIG. 8. In the present exemplary embodiment, at least one of athree-dimensional tomographic image and a three-dimensional motioncontrast image is saved for one examination. Further, in a case where anen-face image has been already generated, one examination includes, forexample, an en-face image and a generation condition representing adepth range of the en-face image in a three-dimensional image. Thus, inthe case where an en-face image is generated, the generation conditionof the en-face image is associated with an examination including theen-face image and a three-dimensional image that is a source of theen-face image.

In step S600, an operator selects an examination. The display controlunit 510 displays, for example as illustrated in FIG. 7, a B-scan imagecorresponding to the selected examination in the display area 130.Further, the display control unit 510 displays an en-face imagegenerated from a three-dimensional image using a predeterminedgeneration condition (a reference surface, a method for determining arepresentative value, and an artifact reduction processing) in thedisplay area 140. More specifically, the en-face image generation unit504 applies the predetermined generation condition stored in a memory orthe like to the three-dimensional image included in the selectedexamination to generate an en-face image. The display control unit 510causes the display apparatus 101 to display the generated en-face imageon the display area 140. The en-face image to be displayed in thedisplay area 140 may be generated from a three-dimensional motioncontrast image or from a three-dimensional tomographic image.

In the present exemplary embodiment, the predetermined generationcondition is used as the generation condition of the en-face image to bedisplayed in the selection of an examination by the operator, but thepresent invention is not limited to this. The operator may specify thegeneration condition in advance. Alternatively, a generation conditionthat is once set by the operator may be recorded in a recording device(not illustrated), and may be read automatically from the recordingdevice to be applied during generation of an en-face image. In thepresent exemplary embodiment, the predetermined generation condition is,for example, that two reference surfaces are an internal limitingmembrane and a pigmented layer of retina, a representative value is anaverage value between the reference surfaces, and the artifact reductionprocessing is not executed. In other words, the generation conditionincludes at least one of information representing a depth range in athree-dimensional image for generating an en-face image (e.g., positioninformation about the two reference surfaces), a method for determininga representative value in a region lying between the two referencesurfaces (depth range), and whether the artifact reduction processing isexecuted. An example of the artifact reduction processing includesprocessing for reducing projection artifact.

In step S610, the operator adjusts the generation condition of adisplayed en-face image. For example, the operator changes the referencesurfaces into a nerve fiber layer and a ganglion cell layer, and arepresentative value into a maximum value between reference surfaces. Ina case where a specific generation condition does not have to bechanged, this step may be omitted.

In a case where the generation condition is changed, the en-face imagegeneration unit 504 generates an en-face image corresponding to thechanged generation condition. The display control unit 510 updates theimage in the display area 140 from the en-face image generated based onthe predetermined generation condition into the en-face image generatedbased on the changed generation condition. A reference surface can bechanged by using a known method. For example, the image processingapparatus 209 displays the reference surfaces 130 a and the 130 b on theB-scan image. The reference surfaces 130 a and 130 b can be acquired insuch a manner that the image processing apparatus 209 analyzes atomographic image to detect a layer boundary. In a case where theoperator selects the reference surface 130 a or 130 b with a mouse andperforms a vertical dragging operation, the image processing apparatus209 moves the selected reference surface vertically to be able to changea position of the reference surface. In the present exemplaryembodiment, the reference surface is moved with the mouse, but thepresent invention is not limited to this. For example, the imageprocessing apparatus 209 detects a plurality of layer boundaries, andthe operator may select any layer boundary from the plurality ofdetected layer boundaries or may specify a straight line horizontal tothe B-scan image. The en-face image generated in step S600 or step S610corresponds to an example of a first en-face image.

In step S620, the operator selects a target to be combined with thedisplayed en-face image. First, the operator selects a menu (notillustrated) in order to execute the combination processing. When themenu is selected, the display control unit 510 causes the displayapparatus 101 to display an examination list 801 to be the combinationtarget on a foreground, for example as illustrated in FIG. 8. Thedisplay control unit 510 causes the respective examinations to bedisplayed in the examination list 801, for example, in a method equal tothe display method for the display area 120. The combination targetselection unit 503 can select whether each of the examinations is thecombination target in accordance with an instruction from the operator.The display control unit 510 causes, for example, an examination 810selected as the combination target to be highlighted and displayed onthe display apparatus 101 to notify the operator of a selected state.For example, as an examination to be the combination target, the displaycontrol unit 510 displays, in the list, an examination in which animaged target eye, an imaging date, and an imaging size are identical tothose in the examination selected in step S600, as an examination to bethe combination target. In other words, the display control unit 510causes the display apparatus 101 to display, as the examination list801, the examination in which the imaged target eye, the imaging date,and the imaging size are the same as those in the examination alreadydisplayed on the display apparatus 101. Each examination is associatedwith identification information about a target eye (e.g., patientidentification information and information representing whether anexamination target is a left eye or right eye), and information such asan examination date and an imaging size. The display control unit 510can cause, based on the identification information or the like, thedisplay apparatus 101 to display the examination in which the imagedtarget eye, the imaging date, and the imaging size are the same as thosein the examination already displayed on the display apparatus 101, asthe examination list 801.

In the present exemplary embodiment, an examination conducted on thesame date as an examination already displayed on the display apparatus101 is displayed, but any examination conducted on the same target eyemay be displayed even if the date is not the same. Further, the displayapparatus 101 displays an examination in the same imaging size as thatin the examination already displayed on the display apparatus 101, butthe configuration is not limited to this. For example, an examinationincluding a grainy image obtained by wide-angle imaging of the sametarget eye, can be the combination target. As a result, alignmentbecomes easy in some cases. Further, an examination in which imaging isdetected as being failed due to blinking during the imaging may be setnot to be displayed and thus does not become the combination target. Forexample, the image processing apparatus 209 can determine based onbrightness of a tomographic image whether imaging is failed due toblinking during the imaging. If the image processing apparatus 209determines that the imaging is failed, information representing failuremay be associated with the examination. The display control unit 510 candetermine, based on the information representing failure, whether theexamination is to be displayed as the examination list 801.

The operator then selects an examination to be the combination target.In the present exemplary embodiment, in an initial state when theexamination list 801 is displayed, an examination imaged in thepanoramic imaging mode is displayed in a state where the examination isselected by the combination target selection unit 503. In other words,the display control unit 510 causes the display unit 101 to display theexamination imaged in the panoramic imaging mode in a state where theexamination is highlighted in the initial state of the examination list801. Therefore, even in a case of the examination in which the imagedtarget eye, the imaging date, and the imaging size are identical tothose in the examination selected in step S600, a display state in theinitial state of the examination list 801 varies in accordance withwhether the examination includes the image captured in the panoramicimaging mode. As a result, in a case of the imaging in the panoramicimaging mode, the operator can avoid the trouble of selecting theexamination as the combination target. Herein, the panoramic imagingmode is an imaging mode in which a plurality of different portions of aneye part is imaged for generating a panoramic image. For example, thisis a mode in which a display position of the fixation lamp issequentially changed automatically or manually so that an image suitablefor the panoramic image can be acquired, and the imaging is carried outat the display positions of the fixation lamp. Therefore, a plurality ofthree-dimensional images captured in the panoramic imaging modecorresponds to an example of a three-dimensional image acquired byimaging areas of a target eye at least partially different.

The information representing the imaging in the panoramic imaging modeis associated with a plurality of examinations in which the imaging iscarried out in the panoramic imaging mode. The combination targetselection unit 503 can select an examination in which the imaging iscarried out in the panoramic imaging mode in the initial state of theexamination list 801, based on the information representing the imagecaptured in the panoramic imaging mode. Thus, the display control unit510 can cause the display unit 101 to display the examination in whichthe imaging is carried out in the panoramic imaging mode in a statewhere the examination is highlighted in the initial state of theexamination list 801, based on the information representing the imagecaptured in the panoramic imaging mode. Only in a case where theexamination selected in step S600 is imaged in the panoramic imagingmode, in the initial state of the examination list 801, the examinationin which the imaging is carried out in the panoramic imaging mode may behighlighted and display on the display unit 101. If the examinationselected in step S600 is not imaged in the panoramic imaging mode, inthe initial state of the examination list 801, the examination in whichthe imaging is carried out in the panoramic imaging mode may not behighlighted.

In a case where the panoramic imaging is carried out more than once, theinformation representing the images captured in the panoramic imagingmode may be associated with the examinations, respectively, so that eachpanoramic imaging can be distinguished. The display control unit 510 maycause the display unit 101 to display each examination in theexamination list 801 with each examination being highlighted so that thepanoramic imaging carried out more than once can be distinguished fromeach other. For example, the display control unit 510 highlights aplurality of examinations relating to a first panoramic imaging using aframe of red, and highlights a plurality of examinations relating to asecond panoramic imaging using a frame of any color other than red. Thehighlighting method is not limited to the above described example aslong as the panoramic imaging carried out more than once can bedistinguished from each other.

In the present exemplary embodiment, the display method for theexamination list 801 is identical to the display method for the displayarea 120 but not limited thereto as long as respective imaging regionsof the examinations can be distinguished from each other. For example,the display control unit 510 can cause the display apparatus 101 todisplay an en-face image generated based on the generation condition setin step S610 or a scanning laser ophthalmoscope (SLO) image.Alternatively, a B-scan image may be set not to be displayed. As aresult, the operator can select a combination target while checking animage state of an examination as the combination target.

After selecting the examination as the combining object, in step S630,the operator selects a button 802 for notifying the image processingapparatus 209 of the start of the combination processing. If the button802 is selected, the en-face image generation unit 504 acquires apredetermined generation condition of an en-face image or the generationcondition of the en-face image changed in step S610 from the memory. Forexample, the changed generation condition of the en-face image isrecorded in a memory associated with an examination. The en-face imagegeneration unit 504 corresponds to an example of an acquisition unitthat acquires a generation condition of a first en-face image generatedfrom a first three-dimensional image of a target eye.

The en-face image generation unit 504 automatically applies the acquiredgeneration condition to all three-dimensional images of examinationsselected in step S620 to generate en-face images associated with theexaminations. Herein, the en-face image generated in step S630corresponds to an example of a second en-face image. Specifically, if apredetermined generation condition is not changed in step S610, thepredetermined generation condition is automatically applied to allthree-dimensional images of the examinations selected in step S620 togenerate en-face images associated with the examinations. If thepredetermined generation condition is changed in step S610, the changedgeneration condition of an en-face image is automatically applied to allthree-dimensional images of the examinations selected in step S620 togenerate en-face images associated with the examinations. Thus, theen-face image generation unit 504 corresponds to an example of a firstgeneration unit that applies the generation condition acquired by theacquisition unit to a second three-dimensional image of a target eye togenerate a second en-face image from the second three-dimensional image.More specifically, in a case where a user selects the first en-faceimage displayed on the display unit and the second three-dimensionalimage as a target of a combined image, the en-face image generation unit504 corresponding to an example of the first generation unit applies thegeneration condition acquired by the acquisition unit to the secondthree-dimensional image of the target eye. In such a manner, the en-faceimage generation unit 504 generates the second en-face image.

Position information about the two reference surfaces included in thegeneration condition is, for example, information representing a name ofa layer boundary. The image processing apparatus 209 applies a knownlayer boundary extracting technique to all the three-dimensional imagesof the examinations selected in step S620. In such a manner, the imageprocessing apparatus 209 can recognize the layer boundary. Accordingly,the en-face image generation unit 504 applies the position informationabout the two reference surfaces relating to the displayed en-face imageto all the images of the examinations selected in step S620. In such amanner, the en-face image generation unit 504 can generate the en-faceimage. Further, the image processing apparatus 209 may align thethree-dimensional image as a source of the displayed en-face image withthe three-dimensional images included in all the examinations selectedin step S620. The en-face image generation unit 504 applies the positioninformation about the two reference surfaces relating to the displayeden-face image to the images of the examinations selected in step S620,based on an alignment result. In such a manner, the en-face imagegeneration unit 504 also can generate the en-face image.

The two-dimensional combined image generation unit 505 then aligns theplurality of generated en-face images to generate a two-dimensionalpanoramic combined image acquired by combining the plurality of en-faceimages, based on an alignment result. More specifically, thetwo-dimensional combined image generation unit 505 generates atwo-dimensional panoramic combined image acquired by combining theen-face image generated in step S600 or S610 with the en-face imagegenerated in step S630. In other words, the two-dimensional combinedimage generation unit 505 corresponds to an example of a secondgeneration unit that generates a combined image by combining the firsten-face image with the second en-face image.

Further, the display control unit 510 displays the generatedtwo-dimensional panoramic combined image in the display area 111.

In the present exemplary embodiment, the operator selects theexamination to be the combined target in step S620, but theconfiguration is not limited to this. For example, in step S610, theoperator who has adjusted the generation condition for an en-face imagemay directly select the start of the combination processing in stepS630. In this case, the combination target selection unit 503, forexample, automatically selects the examination in which the imaging iscarried out in the panoramic imaging mode. The two-dimensional combinedimage generation unit 505 can generate a two-dimensional panoramiccombined image similarly to the above-described method.

In the present exemplary embodiment, as the generation condition of anen-face image to be applied automatically to all three-dimensionalimages to be combined, the generation condition including two referencesurfaces, the representative value determining method, and whether theartifact reduction processing is executed is used. However, thegeneration condition is not limited to this condition. For example, thegeneration condition may not include whether the artifact reductionprocessing is applied. Further, for example, the generation conditionmay include luminance, contrast, a gamma curve, and a noise reductionmethod to be adjusted for causing an en-face image suitable for display.As a result, a sense of incongruity felt when a plurality of images withdifferent luminance and contrasts is combined can be reduced.

In the present exemplary embodiment, only one en-face image is displayedon the display apparatus 101, but the display is not limited to this.The generation condition may be applied to, for example, an imageprocessing apparatus that simultaneously displays en-face imagesgenerated under the generation conditions of a plurality of en-faceimages. In this case, the generation condition of an en-face image to bepreferentially used for the combination processing is determined inadvance, so that a two-dimensional panoramic combined image can begenerated similarly to the above-described method. For example, theen-face image generation unit 504 applies the generation conditionrelating to an en-face image clicked (selected) from the plurality ofen-face images displayed on the display apparatus 101 to all the imagesof the examinations selected in step S620. In such a manner, the en-faceimage generation unit 504 generates an en-face image.

According to the present exemplary embodiment described above, thegeneration condition of an en-face image specified for onethree-dimensional image is automatically applied to all thethree-dimensional images to be combined so that a two-dimensionalpanoramic combined image can be acquired. As a result, time and efforttaken when the generation condition of an en-face image is set for thethree-dimensional images to be combined can be simplified.

Further, as described in the present exemplary embodiment, when atwo-dimensional panoramic combined image is acquired, en-face images aregenerated in advance and are combined. As a result, alignment andcombining of three-dimensional images are not necessary, and thus theprocessing can be sped up.

The above-described method enables a two-dimensional combined image tobe easily acquired from a plurality of three-dimensional images at ahigh speed.

A second exemplary embodiment will describe an example of apredetermined generation condition of an en-face image or an example ofa case where in step S610, an operator sets the generation condition ofan en-face image that the reference surfaces are retinal pigmentepithelium and a lower end of an image, a representative value is amaximum value between the reference surfaces, and the artifact reductionprocessing is executed.

For example, in a motion contrast image of a portion lower than theretinal pigment epithelium, projection artifacts easily occur. Theprojection artifacts occurs in such a manner that a blood vesselidentical to a blood vessel on an upper layer is drawn as a false bloodvessel, or a shadow of the blood vessel on the upper layer is drawn in amotion contrast image of a periphery of an area with low luminance suchas an external granular layer. For this reason, for example, in stepS610, if an area lower than the retinal pigment epithelium (area on achoroid side) is specified as the generation condition of an en-faceimage, it is preferable that processing for reducing a projectionartifact is executed. This processing can employ known various methodsfor correcting information about a blood vessel on a lower layer basedon the information about the blood vessel on the upper layer.

FIG. 9 is a flowchart illustrating an example of an operation of theimage processing apparatus 209 according to the second exemplaryembodiment after the operator selects the button 802 in step S630. Sincethe operation flow for the operator and the configuration of the imageprocessing apparatus 209 are similar to the operation flow and theconfiguration in the first exemplary embodiment, description thereof isomitted.

In step S900, for example, the en-face image generation unit 504generates alignment en-face images with respect to the three-dimensionalimage of the examination selected in step S600 and all thethree-dimensional images of the examinations selected in step S620. Thealignment en-face images are used for the alignment to be carried outunder such a condition that internal limiting membrane and a surfacelower from a boundary between a ganglion cell layer and an innerplexiform layer by 50 μm are reference surfaces. The reference surfacesof the en-face images for the alignment are not limited to theabove-described surfaces, and for example, any reference surfacesincluding the ganglion cell layer may be used. This is because acapillary network of a surface layer is present on the ganglion celllayer and the capillary network of the surface layer can be utilized fora reference of the alignment. In such a manner, the en-face imagegeneration unit 504 corresponding to an example of the first generationunit generates a third en-face image and a fourth en-face image from afirst three-dimensional image and a second three-dimensional image. Thethird and fourth en-face images are within a depth range including anarea on a vitreous body side with respect to a depth range in the firstthree-dimensional image included in the generation condition. The thirden-face image and the fourth en-face image correspond to an example ofthe en-face images for the alignment.

Further, the en-face images for alignment may be always generated.Alternatively, if the depth range represented by the predeterminedgeneration condition of an en-face image or the generation conditionchanged in step S610 does not include the ganglion cell layer, theen-face image generation unit 504 may generate en-face images includingthe ganglion cell layer as the en-face images for alignment. Morespecifically, if the depth range in the first three-dimensional imageincluded in the generation condition does not include the ganglion celllayer, the en-face image generation unit 504 corresponding to an exampleof the first generation unit generates the third en-face image and thefourth en-face image. If the depth range in the first three-dimensionalimage included in the generation condition includes the ganglion celllayer, the en-face image generation unit 504 corresponding to an exampleof the first generation unit does not generate the third en-face imageand the fourth en-face image.

In step S910, the en-face image generation unit 504 generates en-faceimages to which the specified generation condition or the generationcondition for an en-face image specified in step S610 is applied.

In step S901, the two-dimensional combined image generation unit 505aligns the plurality of generated en-face images for alignment. Thetwo-dimensional combined image generation unit 505 aligns the pluralityof en-face images for alignment based on, for example, a blood vesselincluded in the en-face images for alignment. Thus, the two-dimensionalcombined image generation unit 505 acquires a misalignment amount of theplurality of en-face images for alignment.

In step S911, the two-dimensional combined image generation unit 505aligns the plurality of en-face images generated in step S910, based onthe alignment result in step S901. More specifically, thetwo-dimensional combined image generation unit 505 aligns the pluralityof en-face images generated in step S910, based on the misalignmentamount acquired in step S901.

In step S920, the two-dimensional combined image generation unit 505generates a two-dimensional panoramic combined image based on thealignment result in step S910. In other words, the two-dimensionalcombined image generation unit 505 corresponding to an example of thesecond generation unit aligns the first en-face image and the seconden-face image to generate a combined image based on the alignment resultbetween the third en-face image and the fourth en-face image.

In step S930, the display control unit 510 causes the display apparatus101 to display the generated two-dimensional panoramic combined image.

In the motion contrast image, a blood vessel on an upper layer such as aretina superficial layer is easily drawn, whereas a blood vessel on alower layer such as a choroid coat is not drawn due to a noise in somecases. For this reason, as in the present exemplary embodiment, en-faceimages on which the blood vessel on the upper layer is drawn are usedfor the alignment between the en-face images. Therefore, the alignmentcan be carried out accurately regardless of the predetermined generationcondition of an en-face image or the generation condition of an en-faceimage specified in step S610.

In the present exemplary embodiment, the en-face images on which theblood vessel on the retina superficial layer is drawn are used for thealignment, but the en-face images are not limited to them. For example,even if the above-described integrated images or SLO images are used forthe alignment, the equivalent effect can be obtained.

According to the present exemplary embodiment described above, thetwo-dimensional panoramic combined image which is aligned accurately andhas desirable reference surfaces can be generated using a simple method.

In the description, the present exemplary embodiment is applied to thecase where the generation condition is such that the reference surfacesare retinal pigment epithelium and the lower end of the image, therepresentative value is the maximum value between the referencesurfaces, and the artifact reduction processing is executed. However,the case to which the present exemplary embodiment is applied is notlimited to the above case of the generation condition. For example, in acase where the depth range included in the generation condition does notinclude the ganglion cell layer, the en-face images for alignment may begenerated.

The first exemplary embodiment has described the case where thetwo-dimensional panoramic combined image is generated as the combinedimage. However, a third exemplary embodiment will describe a case wherean image is generated as the combined image by averaging a plurality ofen-face images generated from a plurality of three-dimensional images ofan approximately the same portion.

A tomographic image captured by the OCT image capturing apparatus 201and a motion contrast image generated based on the tomographic imageincludes a noise component such as a speckle noise in some cases. As anoise reduction method, an average value filtering method is known, butif averaging is performed in one image by using an average value filter,there may be a worry that a resolution of the image is deteriorated. Onthe other hand, since the speckle noise is generated randomly, averaginga plurality of en-face images obtained by imaging an approximately sameportion enables the noise to be reduced while maintaining the resolutionof the image. In the present exemplary embodiment, this processing isreferred to as combination processing.

The OCT image capturing apparatus 201 according to the present exemplaryembodiment has a same portion imaging repetition mode as a mode forcontinuously acquiring tomographic images of an approximately sameportion of a target eye. If the operator who desires to image thepartial region 301 (see FIG. 3) of the target eye selects this mode, theOCT image capturing apparatus 201 does not change an imaging conditionsuch as the position of the internal fixation lamp and an imaging sizeafter the imaging of the partial region 301 is completed. Thus, thepartial region 301 can be again imaged under the same condition. Thetomographic image acquisition unit 501 can acquire a plurality oftomographic images that approximately matches with the partial region301 by the OCT image capturing apparatus 201 repeating the imagingdesired number of times.

The operation to be performed by the operator according to the presentexemplary embodiment is similar to the operation according to the firstexemplary embodiment. The operation of the image processing apparatus209 corresponding to the operation to be performed by the operator willbe described.

In step S600, when the operator selects a menu (not illustrated), inorder to execute the combination processing, the display control unit510 causes the display apparatus 101 to display the examination list 801to be the combination target on a foreground similarly to the firstexemplary embodiment. Further, in step S600, one examination isselected, and thus an en-face image or the like is displayed on thedisplay apparatus 101. As the examination to be the combination target,the display control unit 510 causes the display apparatus 101 todisplay, in form of a list, examinations in which a target eye identicalto that in the examinations selected in step S600 is imaged. Theoperator then selects an examination to be the combing target from theexamination list 801. In the present exemplary embodiment, in an initialstate where the examination list 801 is displayed, the display controlunit 510 displays examinations in which the imaging is performed in thesame portion imaging repetition mode in a state that the combinationtarget selection unit 503 is selecting the examination. As a result, ifthe imaging is performed in the same portion imaging repetition mode,time and effort taken by the operator to select the examinations to bethe combination targets can be reduced. The plurality of examinations inwhich the imaging is performed in the same portion imaging repetitionmode is associated with information representing the imaging in the sameportion imaging repetition mode. The combination target selection unit503 can select the examinations in which the imaging is performed in thesame portion imaging repetition mode in the initial state of theexamination list 801, based on the information representing the imagingin the same portion imaging repetition mode. In other words, the displaycontrol unit 510 can highlight the examinations in which the imaging isperformed in the same portion imaging repetition mode to cause thedisplay unit 101 to display the examinations in the initial state of theexamination list 801, based on the information representing the imagingin the same portion imaging repetition mode.

The operator who has selected the examinations to be the combinationtargets selects, in step S630, the button 802 for notifying the imageprocessing apparatus 209 of the start of the combination processing.After the selection of the button 802, the en-face image generation unit504 automatically applies the predetermined generation condition forgenerating an en-face image or the generation condition for an en-faceimage specified in step S610 to all images of the examinations selectedin step S620. In such a manner, the en-face image generation unit 504generates the en-face images corresponding to the respectiveexaminations. The two-dimensional combined image generation unit 505then aligns the plurality of generated en-face images, and combines theplurality of en-face images to generate a two-dimensional combined image(averaged image) based on the alignment result. Further, the displaycontrol unit 510 displays the generated two-dimensional combined imageon the display area 111. The image generated in such a manner can beexpected to have less noise than each of the images of the examinations.

According to present exemplary embodiment described above, thetwo-dimensional combined image where noise of the plurality ofthree-dimensional images are reduced can be acquired at high speed usinga simple method.

A fourth exemplary embodiment will describe an example of the operationof the image processing apparatus 209 in a case where after the operatoracquires the two-dimensional combined image in step S630, the processingreturns to step S610 again, and en-face images are generated under adifferent generation condition.

Examples of the operating method and the operation of the imageprocessing apparatus 209 according to the present exemplary embodimentwill be described with reference to FIGS. 10 and 11. As illustrated inFIG. 10, the display apparatus 101 according to the present exemplaryembodiment displays the panoramic combined image display area 111, thecaptured image list display area 120, the selected examination B-scanimage display area 130, and the en-face image display area 140 on onescreen. Since the processing in steps S600 to S630 in FIG. 11 to beexecuted first are similar to the processing in the first exemplaryembodiment, detailed description thereof is omitted. At least one of theexamination (three-dimensional image) selected in step S620 andinformation representing the misalignment of the plurality of en-faceimages acquired in step S630 is stored in the memory (not illustrated)of the image processing apparatus 209.

First, the processing after step S630 will be described. If the operatorobserves the combined image generated in step S630 and changes thegeneration condition of the en-face images used for combining, theprocessing returns to step S610, and the operator again adjusts thegeneration condition of the en-face images.

The operator who has adjusted the generation condition selects a menu(not illustrated) in step S630 in order to execute the combinationprocessing again. This menu includes a user interface (UI) that caninstruct the recombining. In the present exemplary embodiment, theoperator can select the recombining, to execute the combinationprocessing without reselecting a combination target examination on theexamination list 801. For example, if the recombining is instructed, thecombination target selection unit 503 acquires an examination(three-dimensional image) selected first in step S620 from the memory.The en-face image generation unit 504 then regenerates en-face imagesusing the adjusted combining condition for images of combination targetexaminations identical to the images in the first combinationprocessing. At this time, the combination target examinations includethe examination selected in step S600. The two-dimensional combinedimage generation unit 505 aligns the plurality of en-face images. Thetwo-dimensional combined image generation unit 505 can execute theprocessing based on the information representing the misalignment of theplurality of en-face images acquired first in step S630. As a result,the time for the alignment processing can be shortened.

After the alignment processing, the two-dimensional combined imagegeneration unit 505 generates a two-dimensional panoramic combined imageor an averaged image similarly to the first processing. The displaycontrol unit 510 displays the generated combined image in the displayarea 111 and saves the combined image on the memory (not illustrated).

The present exemplary embodiment can produce an effect similar to aneffect of the first exemplary embodiment. Further, the second andsubsequent image selection and alignment processing in the combinationtarget processing are executed based on the results of the first imagesection and alignment processing, so that the processing time can beshortened.

The above-described first to fourth exemplary embodiments may besuitably combined. For example, the second and third exemplaryembodiments may be combined. More specifically, the second exemplaryembodiment may be applied to the alignment in a case where an imageacquired by averaging the plurality of en-face images is generated as anexample of the combined image other than the panoramic combined image.

Further, the second exemplary embodiment may be applied to the fourthexemplary embodiment. For example, the information representing themisalignment of the plurality of en-face images acquired first in stepS630 may be acquired by the en-face images including the ganglion celllayer.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-171188, filed Sep. 6, 2017, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a firstgeneration unit configured to generate (a) a first en-face image byapplying a generation condition to a first three-dimensional image of asubject, (b) a second en-face image by applying the generation conditionto a second three-dimensional image of the subject, (c) a third en-faceimage using a part of the first three-dimensional image, and (d) afourth en-face image using a part of the second three-dimensional image,wherein the third en-face image and the fourth en-face image correspondto a depth range located closer to a vitreous body side than a depthrange included in the generation condition; and a second generation unitconfigured to generate a combined image by using the first en-faceimage, the second en-face image, and a result of alignment between thethird en-face image and the fourth en-face image.
 2. The imageprocessing apparatus according to claim 1, wherein the first generationunit generates, in a case where the first en-face image displayed on adisplay unit is selected by a user and the second three-dimensionalimage is selected as a target of the combined image, the second en-faceimage by applying the generation condition to the secondthree-dimensional image of the subject.
 3. The image processingapparatus according to claim 1, wherein the generation conditionincludes at least one of a depth range of the first en-face image in thefirst three-dimensional image, a method for determining a representativevalue of a pixel in the depth range, and whether artifact reductionprocessing is executed.
 4. The image processing apparatus according toclaim 1, wherein the first three-dimensional image and the secondthree-dimensional image are three-dimensional images obtained by imagingareas of the subject at least different in part, and wherein thecombined image is a panoramic image of the first en-face image and thesecond en-face image.
 5. The image processing apparatus according toclaim 1, wherein the first three-dimensional image and the secondthree-dimensional image are obtained by imaging areas of approximatelysame portions of the subject, and wherein the combined image is obtainedby averaging the first en-face image and the second en-face image. 6.The image processing apparatus according to claim 1, wherein the secondgeneration unit aligns the first en-face image with the second en-faceimage using the result of alignment between the third en-face image andthe fourth en-face image to generate the combined image.
 7. The imageprocessing apparatus according to claim 6, wherein the first generationunit generates, in a case where the depth range in the firstthree-dimensional image included in the generation condition does notinclude a ganglion cell layer, the third en-face image and the fourthen-face image.
 8. The image processing apparatus according to claim 7,wherein the first generation unit does not generate, in a case where thedepth range in the first three-dimensional image included in thegeneration condition includes the ganglion cell layer, the third en-faceimage and the fourth en-face image.
 9. The image processing apparatusaccording to claim 1, wherein the three-dimensional image is athree-dimensional tomographic image obtained by using optical coherencetomography.
 10. The image processing apparatus according to claim 1,wherein the three-dimensional image is a three-dimensional motioncontrast image based on a tomographic image obtained by using opticalcoherence tomography.
 11. The image processing apparatus according toclaim 1, wherein the subject is an eye, and wherein the depth range isobtained using a result of detection of a layer boundary in the firstthree-dimensional image and the second three-dimensional image.
 12. Asystem comprising: an optical coherence tomography image capturingapparatus that includes a detector configured to detect, as aninterference signal, interference light obtained by combining returnlight and reference light, the return light returning from a subjectirradiated with measurement light; and the image processing apparatusaccording to claim 1, further comprising an obtaining unit configured toobtain the first three-dimensional image and the secondthree-dimensional image by using the detected interference signal. 13.An image processing method comprising: generating (a) a first en-faceimage by applying a generation condition to a first three-dimensionalimage of a subject, (b) a second en-face image by applying thegeneration condition to a second three-dimensional image of the subject,(c) a third en-face image using a part of the first three-dimensionalimage, and (d) a fourth en-face image using a part of the secondthree-dimensional image, wherein the third en-face image and the fourthen-face image correspond to a depth range closer to a vitreous body sidethan a depth range included in the generation condition; and generatinga combined image by using the first en-face image, the second en-faceimage, and a result of alignment between the third en-face image and thefourth en-face image.
 14. A non-transitory computer-readable storagemedium storing a program that when run on a computer causes the computerto execute the image processing method according to claim 13.